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ABSTRACT 

This  paper  describes  efforts  to  develop  and  validate  a 
performance  measurement  system  (PMS)  for  C-5A  aircrews.  To  date,  a 
PMS  has  been  developed  and  successfully  integrated  with  one  of  the 
C-5A  flight  simulators  at  Altus  AFB,  OK.  The  system  is  currently 
being  used  as  a  routine  part  of  the  simulator  training  curriculum  and 
is  undergoing  evaluation.  This  paper  describes  the  capabilities  of 
the  simulator  PMS  and  presents  the  results  of  the  preliminary  data 
gathered  during  the  validation  phase  of  the  project. 


Introduction 


In  the  late  70's,  the  need  for  improved  measurement  capabilities  in  the 
operational  training  environment  was  realized  and  a  major  R&D  program  was 
initiated.  In  order,  to  document  the  effectiveness  and  efficiency  of  the 
simulator  training  syllabus,  it  seemed  desirable  to  quantitatively  assess 
aircrew  proficiency  both  in  the  simulator  and  the  aircraft.  The  development 
of  objective  performance  assessment  capabilities  for  operational  flight 
training  systems  was  viewed  as  a  two-phased  process.  The  first  phase  would 
focus  on  the  implementation  of  a  measurement  system  within  the  flight 
simulation  environment,  while  the  second  would  focus  on  the  aircraft.  Two 
target  applications  were  eventually  selected:  first,  the  development  of  a  PMS 
for  the  C-5A  transport  aircraft,  which  is  the  focus  of  the  present  paper  and 
second,  the  development  of  an  air  combat  maneuvering  PMS  applicable  to  both 
the  simulation  environment  and  instrumented  range  facilities. 

Description  of  C-5A  PMS 

DMC  5  front -end  analysis  was  initiated  to  define  requirements  for  an  objective 
PMS  for  members  of  the  C-5A  aircrew  utilizing  the  existing  C-5A  simulator. 

i  in™?11  functional  capabilities  described  in  the  final  report  (Swink  et 
aj_,  1978),  a  contract  was  awarded  for  the  development,  fabrication,  and 
integration  of  a  measurement  system  for  the  C-5A  flight  simulator.  The 
resulting  system  was  installed  on  one  of  the  simulators  located  at  the  C-5A 
Training  Squadron  at  Altus  AFB  OK.  Acceptance  testing  was  completed  in 
October  1982.  Currently,  the  system  is  undergoing  a  two-year  evaluation. 

Before  describing  the  evaluation  and  some  of  the  initial  findings,  a  brief 
description  of  its  functional  capabilities  is  presented. 

Mission  Control .  The  C-5A  PMS  provides  the  potential  for  a  precise  definition 
of  mission  profiles  whereby  the  proficiency  of  each  aircrew  member  can  be 
determined.  The  level  of  detail  for  the  profile  corresponds  to  that  of  a 
complete  specification  of  student  responses  and  aircraft/environmental 
factors  (conditions,  standards,  etc.)  to  be  addressed  by  the  system.  The 


163 


Report  Documentation  Page 

Form  Approved 

OMB  No.  0704-0188 

Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 

VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 

1 .  REPORT  DATE  2.  REPORT  TYPE 

APR  1984  Conference  Proceedings 

3.  DATES  COVERED 

01-05-1983  to  31-03-1984 

4.  TITLE  AND  SUBTITLE 

The  Measurement  of  C-5  Aircrew  Performance 

5a.  CONTRACT  NUMBER 

F33615-83-C-0025 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

63327F 

6.  AUTHOR(S) 

Wayne  Waag;  David  Hubbard 

5d.  PROJECT  NUMBER 

2359 

5e.  TASK  NUMBER 

01 

5f.  WORK  UNIT  NUMBER 

23590104 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS (ES) 

Air  Force  Human  Resources  Laboratory, Operations  Training  Division, 6030 
South  Kent  Street, Williams  AFB,AZ, 85212-6061 

8.  PERFORMING  ORGANIZATION  REPORT  NUMBER 

AFHRL/OT 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS  (ES) 

Air  Force  Research  Laboratory,  Human  Effectiveness  Directorate,  Warfighter 

10.  SPONSOR/MONITOR’S  ACRONYM(S) 

AFRL;  RHA 

Readiness  Research  Division,  Mesa,  AZ,  85212-6061 

11.  SPONSOR/MONITOR’S  REPORT  NUMBER(S) 

AFRL-RH-AZ-PR- 1984-0001 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

In  Proceedings  of  the  Ninth  Psychology  in  the  Department  of  Defense  Symposium,  held  18-20  Apr  84,  in  Colorado  Springs  CO, 
pp  163-167  (NOTE:  Entire  conference  proceedings  may  be  found  at  ADA141043) 

14.  ABSTRACT 

This  paper  describes  efforts  to  develop  and  validate  a  performance  measurement  system  (PMS)  for  C-5  A  aircrews.  To  date,  a 

PMS  has  been  developed  and  successfully  integrated  with  one  of  the  C-5A  flight  simulators  at  Altus  AFB,  OK.  The  system  is 
currently  being  used  as  a  routine  part  of  the  simulator  training  curriculum  and  is  undergoing  evaluation.  This  paper  describes 
the  capabilities  of  the  simulator  PMS  and  presents  the  results  of  the  preliminary  data  gathered  during  the  validation  phase  of 
the  project. 

15.  SUBJECT  TERMS 

C-5A;  Aircrew  performance;  Performance  measurement  system;  PMS;  Aircrews;  Aircrew  training;  Flight  simulators;  Military 
psychology;  Flight  testing;  Performance(Human);  Flight  crews;  Measurement;  Training  devices;  Operational  effectiveness, 
Research  management;  Symposia 

16.  SECURITY  CLASSIFICATION  OF:  17.  LIMITATION  OF 

ARQTR ATT 

18.  NUMBER  19a.  NAME  OF  RESPONSIBLE 

OFPAfTFS  PFRSfW 

a.  report  b.  abstract  c.  this  page  Public  Release 

unclassified  unclassified  unclassified 

5 

Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


capability  is  also  provided  for  the  creation  of  new  mission  scenarios  by 
instructor  personnel  within  Military  Airlift  Command  (MAC).  The  C-5A  PMS  has 
the  capability  to  operate  in  either  a  preprogrammed  or  manual  mode.  In  the 
preprogrammed  mode,  the  PMS  does  not  allow  any  alteration  of  the  predefined 
mission;  only  monitoring  and  feedback  controls  are  available.  The  manual 
intervention  mode,  however,  provides  the  instructors  with  the  ability  to 
completely  control  the  PMS  mission.  Predefined  malfunctions  may  be  activated 
or  cleared  at  any  time  and  additional  malfunctions  not  contained  in  the 
original  PMS  scenario  may  be  inserted.  Of  course,  the  instructors  may  allow 
the  PMS  scenario  to  operate  as  initially  designed  by  the  course  developers. 

Performance  Monitoring,  Assessment,  and  Analysis.  The  C-5A  PMS  monitors  most 
switch  positions,  control  settings,  and  instrument  readings  from  the  flight 
deck.  It  provides  a  real-time  evaluation  of  proficiency  for  all  crew  members 
individually,  as  well  as  the  coordination  among  aircrew  members.  Proficiency 
evaluations  can  be  accomplished  using  the  preprogrammed  mission  profiles.  It 
requires  that  mission  essential/critical  flight  tasks  be  included  within  the 
preprogramed  profiles  and  that  performance  standards  be  precisely  defined. 

The  system  provides  a  measurement  capability  for  checklists,  procedures, 
navigational  profiles,  and  aircraft  state  parameters.  It  also  has  the 
capability  to  store,  retrieve,  and  summarize  all  performance  measures. 

Performance  Feedback  and  Displays.  The  PMS  provides  the  following  display  and 
feedback  capabilities:  (a)  mission  sequence  display— summary  displays  of  the 
sequences  of  tasks;  (b)  route  chart  display— graphic  background  displays 
corresponding  to  departure,  enroute,  and  approach  plates;  (c)  checklist/ 
procedure  display— displays  of  predefined  sequences  of  actions  to  be  performed 
by  crew  members;  (d)  error  alert  display— message  alerting  instructor  to  crew 
errors  as  they  occur  in  the  predefined  tasks;  (e)  proficiency  assessment 
display— detailed  alphanumeric  displays  relative  to  any  specific  predefined 
performance  segment  or  task;  (f)  debriefing  report— hard-copy  containing 
objective  performance  data  which  the  instructor  may  use  for  debriefing;  and 
(g)  help  display — 22  pages  of  on-line  instruction  on  the  use  of  the  PMS. 

To  meet  these  functional  capabilities,  a  "piggy-back"  system  configuration 
was  required.  In  other  words,  the  C-5A  PMS  is  autonomous  and  independent  of 
the  host  simulator  in  the  sense  of  using  its  own  processor  and  peripherals. 

All  mission  control,  measurement  computations,  and  display  capabilities  are 
accomplished  with  the  PMS  hardware.  It  is  interfaced  with  the  host  simulator, 
so  that  it  passively  monitors  all  I/O  signals  (thereby  obtaining  all  necessary 
data).  It  also  has  the  capability  to  pass  data  actively  to  the  host  simulator 
for  problem  control  and  malfunction  insertion,  as  necessary. 

From  this  brief  description,  it  should  be  apparent  that  the  C-5  PMS 
provides  a  much  wider  range  of  capabilites  than  only  performance  assessment. 

It  encompasses  most  functions  which  have  been  typically  associated  with  the 
design  of  instructor /operator  stations.  As  such,  it  can  be  considered  a 
stand-alone  instructional  support  system.  Discussion  now  turns  to  the 
evaluation  of  the  C-5A  PMS  and  some  of  the  initial  findings.  - 

Evaluations  Plan  for  the  C-5A  PMS 

The  evaluation  of  the  C-5A  PMS  will  address  the  following  areas:  mission 
generation,  operation,  measurement,  perceived  operational  utility,  and 
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operational  suitability. 


Scenario  Pesign  Capabilities.  The  PMS  scenario  author  has  a  number  of 
difficult  tasks.  He  must  specify  all  relevant  mission  components  in 
comprehensive  detail.  Flight  checklists  must  be  specified  not  only  as  to 

UidS  t(?  s®?ue"ce*  Navigational  profile  information  must  be 
included  for  the  main  flight  path  and  any  alternative  flight  paths  among  which 
wish  the  students  to  choose.  The  PMS  author  is  presently 
items.  Different  forms  are  provided  to  aid  in  constructing 
files  that  constitute  a  PMS  scenario.  A  line  text  editor 
with  the  means  of  entering  data  from  these  forms  into 
it  once  it  has  been  entered.  The  mission  generation 

and  consistency  check  of  the  work.  A  consideration 
to  the  following  questions:  Are  the  current  aids 
If  they  need  improvement,  how  should 
aids  needed?  If  so,  what  are  they  and 


the  author  might 
aided  by  several 
the  various  disk 
provides  the  author 
system  and  modifying 
program  provides  a  syntax 
of  this  complex  job  leads 
adequate  or  do  they  need  improvement? 
they  be  improved?  Are  any  additional 


how  should  they  be  integrated  into  the  system? 


System  Operation.  The  PMS  operation  consists  of  two  major  types  of 
operations:  housekeeping  and  mission  conduct.  The  housekeeping  operations 
consist  of  the  daily  turn-on  procedure,  the  confidence  test,  and  the  system 
shutdown.  The  mission  conduct  operations  consist  of  commands  to  perform  among 
other  things  the  following:  selection  of  PMS  operating  mode,  performance 
monitoring,  sign-on,  real-time  interaction  with  PMS  monitoring,  instructor 
station  commands,  and  program  termination.  Of  these  two  types  of  operations, 
it  is  the  mission  conduct  operations  that  have  the  greatest  impact  upon  the 
C-5A  instructors.  Another  objective  will  be  to  assess  these  operations  in 
order  to  identify  the  following:  operations  that  are  confusing  or  difficult 
to  perform;  operations  that  are  unused  or  ineffective;  operations  that  are  not 
presently  in  the  system  that  could  prove  helpful  to  instructors;  suggestions 
for  improving  or  replacing  operations  that  are  presently  confusing  or 
difficult  to  perform;  and  operations  that  instructors  find  helpful  or 
effecti ve. 


Measurement  Validation.  At  present,  the  PMS  provides  performance  measures  at 
five  different  levels.  Level  1  measurement  consists  of  assessment  for 
individual  tasks  (e.g.,  lowering  the  landing  gear  before  final  approach). 

Each  task  is  assigned  a  point  value  and  tasks  are  scored  according  to  a 
predefined  algorithm.  There  are  separate  scores  on  each  task  for  each  crew 
member  on  each  individual  task.  Scored  tasks  have  been  specified  in  advance 
by  the  scenario  author  along  with  the  total  possible  point  value  for  each 
task.  The  individual  scores  are  then  combined  into  three  performance  task 
groups  for  level  2  scores:  check  list /procedures,  monitorable  parameters,  and 
navigational  profiles.  Level  2  scores  are  computed  by  taking  the  total  points 
earned  under  each  group.  If  certain  critical  tasks  are  missed,  the  overall 
total  is  reduced  by  a  predetermined  criticality  factor.  Level  2  scores  are 
obtained  for  each  crew  member  for  each  of  the  three  performance  tasks  groups. 
In  addition.  Level  2  scores  are  obtained  for  crew  coordination  in  each  of  the 
three  perforamance  groups.  If  the  scenario  has  two  sessions,  a  separate  set 
of  Level  2  scores  are  computed  from  the  Level  1  scores  of  the  second  session. 
Level  3  scores  are  computed  for  each  crew  member  and  crew  coordination  from 
the  Level  2  performance  measurement  group  scores.  In  computing  Level  3 
scores,  each  individual  Level  2  score  is  first  multiplied  by  a  predetermined 
weight  and  then  summed.  The  total  score  is  then  reduced  by  the  appropriate 
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criticalty  factors.  The  Level  3  scores  are  combined  in  a  like  manner  to 
obtam  an  overall  Level  4  score  for  each  of  the  two  scenario  sessions  The 

mission?  4  SC°reS  are  combined  for  a  single  Level  5  score  for  the  entire 


As  part  of  the  test  of  the  validity  of  the  PMS  assessment  nroceriurpe 
analysis  will  be  conducted  to  see  how  Level  2  and  higher  scores  relate  to 
instructor  evaluations  of  crew  performance.  In  addition ,  e?perienced  f^oht 

ISSar,  Crews  W1j1  ^  compared  with  inexperienced  flight  student  crews 
hese  PMS  measures  to  determine  if  the  scores  can  discriminate  between 
experienced  and  inexperienced  personnel.  Since  the  assignment  of  weight  and 
rV  h+a  ^  involves  a  certain  amount  of  subjectivity,  different 

then^P^P^pH^03  Ui  ^Ct0rs  "in  be  used  ™  scoring.  These  new  scores  will 
"  e  te^d  for  relation  with  instructor  evaluation  and  discrimination 
ability.  Tfiis  evaluation  should  help  to  determine  how  much  care  is  needed  in 
assigning  weight  and  criticality  factors  to  the  various  levels  of  scores 


Operational  Utility.  The  PMS  is  designed  as  a  teaching  aid  for  C-5A 
instructors.  There  are  four  basic  teaching  aids  performed  by  the  PMS- 
performance  feedback  to  the  student,  objective  mission  assessment,  mission 
monitoring  for  the  instructor,  and  mission  tailoring  to  student  needs. 
Performance  feedback  is  provided  for  the  student  in  the  form  of  a  printed 
debriefing  report.  The  report  indicates  which  tasks  were  incorrectly 
performed,  which  flight  parameters  were  out  of  bounds,  etc.  The  report  also 
contains  the  five  levels  of  scoring  as  an  objective  assessment  of  the 
mission.  During  a  simulated  mission,  the  PMS  provides  the  instructor  a 
display  which  alerts  him  as  soon  as  a  student  error  is  committed.  The  display 
also  provides  the  means  to  monitor  the  progress  of  the  mission,  the  course 
heading,  the  values  of  various  parameters,  etc.  In  the  manual  mode,  the 
instructor  has  the  freedom  to  alter  the  mission  by  inserting  or  clearing 
various  simulated  aircraft  malfunctions,  change  selected  simulated  atmospheric 
conditions,  suspend  the  session,  skip  over  portions  of  the  scenario,  repeat 
portions  of  the  scenario,  etc.  These  instructor  options  provide  the 
instructor  with  the  ability  to  individualize  the  session  to  the  students. 


In  order  to  determine  the  perceived  utility  of  the  PMS  in  the  C-5A 
training  program,  the  reaction  of  the  instructors  and  students  to  the  system 
is  needed.  In  this  regard,  some  questions  include:  Are  the  PMS  displays  and 
printouts  conveying  the  necessary  information  or  is  there  information  not 
provided  that  would  be  useful?  Is  any  of  the  information  currently  presented 
not  needed?  What  PMS  features  are  used  most  by  the  instructors?  What 
additional  features,  if  any,  would  the  instructors  like  to  have  available  to 
them? 


Operational  Suitability.  Operational  suitability  factors  pertain  to  how  well 
a  device  meets  accepted  equipment  serviceability  requirements  within  its 
intended  operating  and  maintenance  environment.  The  major  question  is:  How 
reliable  is  the  PMS  hardware  in  the  operational  environment.?  In  order  to 
answer  this,  a  maintenance  log  and  an  instructor  discrepancy  log  will  be  kept 
on  the  PMS  during  the  entire  test  and  evaluation  period. 

Preliminary  Findings 

The  accomplishment  of  the  above  objectives  required  that  the  PMS  be  used 
in  the  various  C-5A  training  courses.  Three  selected  missions.  Copilot 
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been  developed  for’ these  missions1  ancfare^ur^  ‘JJSS10ns‘  PMS  scenarios  have 
training  curriculum  at  Altus  AFB.  As  a  resuit^f  3i  part  of  the  C-5A 

informal  observat  ions  have  been' made  relati™*^  J|hiS  ^mentation  several 
the  PMS  as  perceived  'by  the  WA  instructors  operational  utility  of 

faf^Navigati’onal  proffie^dilplays11^  VJ*  '"'“"frntlom  included: 

graphics  prlnte?  has  been  added  io  the  fjs  to  5™?^  IP  respo"se'  a 
track  and  glideslope  displays  for  use  in  debriefing  „  wPue!  °f  the  ground 

i:5l  IPPiSirltiias 

i  Si  ?,_*  .  e  tfie  Parameters  monitoring  capability  of  PMS  Although  thpv 

feel  that  maintenance  of  flight  parameters  is  important  the  aualitv^of  thp 

„^C  ?S’at?r  is  such  that  «  15  impossible  to  Lintain  tLse  wUhln 
tSe  simu1atSneajl2C™i  til  C  W°rdS’  the  Pr°blem  ,ies  w1th  the  <"“»**  °f 

like  thts  capability,  although  the  .8  sec  sampling  rate  of  the  PMS  Is  * 

sometimes  too  slow  to  catch  momentary  switch  positions.  Many  IFEs  report  that 
they  like  having  PMS  back  them  up  when  a  student  c laim  ' to  &e  performed  a 

insertion  ^Mo  IFE  hasn'Lfe"  »«  00  AufiStSTSlfSction 

the  PMS  thpMTFF  fniiJjk  h3^ng  th®  P^S  insert  the  malfunctions.  Without 
tne  PMS,  the  IFE  follows  a  writen  syllabus  that  indicates  what  and  when 

ma  unctions  are  to  be  entered.  The  IFE  must  manually  insert  the  malfunction 
at  the  specified  time  by  throwing  a  switch  or  turning  a  knob  on  the 

Sifr4tM,^„stAr^11IS?1”'-  SUCh  “  1"  oil  pressure. 

someAof0the0in1tial°responsesbt0etheC|JseCl0fntheapMSt!eSThesedperceptionsemay 

change  as  instructors  gain  experience  in  the  use  of  the  system.  Data 

ohJprM^r  f°r  Jhe  e^luation  wil1  continue  through  Dec  84  so  that  all  test 
objectives  can  be  achieved.  At  this  point,  there  seems  little  doubt  that  many 
of  the  capabilities  of  the  PMS  will  be  considered  a  useful  adjunct  to 
aiming.  As  such,  the  product  of  this  development  program  will  lead  to  a 
acquis i t ions ^ 1 C at  1  on  of  requirements  for  such  capabilities  in  future Viator 
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